Evaluating the use of locally-based health facility assessments in Afghanistan: a pilot study of a novel research method

BACKGROUND Through the Balanced Scorecard program there have been independent, annual and nationwide assessments of the Afghan health system from 2004 to 2013. During this period, Afghanistan remained in a dynamic state of conflict, requiring innovative approaches to health service evaluation in insecure areas. The primary objective of this pilot study was to evaluate the reliability of health facility assessments conducted by a novel, locally-based data collection method compared to a standard survey team. METHODS In this cross-sectional study, one standard survey team of clinicians and multiple rapidly trained locally-based survey teams of teachers conducted health facility assessments in Badghis province, Afghanistan from March - August, 2010. Outpatient facilities covered under the country's Basic Package of Health Services were eligible for inclusion. Both approaches attempted to survey as many health facilities as safely possible, up to 25 total facilities per method. Each facility assessed was scored on 23 health services indicators used to evaluate performance in the annual Balanced Scorecard national assessment. For facilities assessed by both survey methods, the indicator scores produced by each method were compared using Spearman's correlation coefficients and linear regression analysis with generalized estimating equations. RESULTS The standard survey team was able to assess 11 facilities; the locally-based approach was able to assess these 11 facilities, as well as 13 additional facilities in areas of greater insecurity. Among the 11 facilities assessed by both approaches, 19 of 23 indicators were statistically similar by survey method (p < .05). Spearman's coefficients varied widely from (-0.39) to (0.71). The differences were greatest for items requiring specialized data collector knowledge on reviewing patient records, patient examination and counseling, and health worker reported satisfaction. CONCLUSIONS This pilot study of a novel method of data collection in health facility assessments showed that an approach using locally-based survey teams provided markedly increased access to areas of insecurity. Though analysis was limited by small sample size, indicator scores used for facility evaluation were relatively comparable overall, but less reliable for items requiring clinical knowledge or when asking health worker opinions, suggesting that alternative approaches may be needed to assess these parameters in insecure environments.